Random Forests Feature Selection with K-PLS: Detecting Ischemia from Magnetocardiograms

Random Forests were introduced by Breiman for feature (variable) selection and improved predictions for decision tree models. The resulting model is often superior to AdaBoost and bagging approaches. In this paper the random forests approach is extended for variable selection with other learning models, in this case Partial Least Squares (PLS) and Kernel Partial Least Squares (K-PLS) to estimate the importance of variables. This variable selection method is demonstrated on two benchmark datasets (Boston Housing and South African heart disease data). Finally, this methodology is applied to magnetocardiogram data for the detection of ischemic heart disease. 1 Partial Least Squares (PLS) and K-PLS Partial Least Squares Regression (PLS) was introduced by Herman Wold [1] for econometrics modeling of multi-variate time series. PLS can be viewed as a “better” Principal Components Analysis (PCA) regression method, where the data are first projected into a different and non-orthogonal basis, and only the most important PLS components (or latent variables) are considered for building a regression model (similar to PCA). The difference between PLS and PCA is that the new set of basis vectors in PLS is not a set of successive orthogonal directions that explain the largest variance in the data, but are actually a set of conjugant gradient vectors to the correlation matrix. The NIPALS implementation of PLS [2] is elegant and fast. Rosipal introduced K-PLS in 2001 [3] as a nonlinear extension to the linear PLS method instead of using linear kernel K-PLS [4]. This nonlinear extension of PLS makes K-PLS a powerful machine learning tool for classification as well as regression. Powerful variable selection methods have been implemented for PLS and K-PLS, and unlike SVMs, multiple output models are easy to implement. K-PLS can also be formulated as a paradigm closely related (and almost identical) [5] to Support Vector Machines (SVM) [6, 7]. K-PLS uses the same kernel trick as is commonly used in SVMs. K-PLS also provides a purely statistical method, that has been widely used in chemometrics during the past decade. In addition, the idea of using of K-PLS rather than SVMs can be motivated on several levels: (i) PLS is the method by choice in chemometrics and drug design, and K-PLS is a natural extension to PLS; (ii) K-PLS results are generally ESANN'2006 proceedings European Symposium on Artificial Neural Networks Bruges (Belgium), 26-28 April 2006, d-side publi., ISBN 2-930307-06-4.